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Questionnaire Design in Broad-Based Evaluation Studies: 
Letting Someone Else Collect Comparison Group Data 1 

in education and social science settings, one of the most commonly used techniques for collecting 
evaluation data is the paper-and-pencii questionnaire. Because the value of the data-and the evaluation 
ft$a&-ts at feast partially dependent on the quality of the data ooDectfen devtee, questionnaire design ban 
integral part of a comprehensive evaluation effort. 

Many fine books on survey research and questionnaire design discuss a wide variety of elements that 
are important to the development of the survey instrument-item wording, selection of response options, 
format of items and response options, Bern order, appearance, etc. (Babbie, 1973; Bardie. Anderson, A 
Nlebuhf, 1986; Fink & Koseooff, 1985; Fowler, 1988; Labaw, 1980). At least one aspect of questionnaire 
design that tends to be overlooked (or given only very general attention) in these books has to do wBh 
considering the use of selected Items from existing sources in order to improve the technical 
characteristics of the Instrument itself or to expand the usefulness of the data collected In the survey. 
Certainly this possibility can be considered to be subsumed in the sections of the aforementioned books 
thai discuss the importance of the survey researcher having weB-defined objectives for his/her survey and 
the importance of designing an instrument that is consistent with those objectives, but such general 
caveats do iittte to alert an interested researcher about alternatives that might already exist or how to 
integrate existing items into new instruments. 

CheDmsky (1985) supports the use of existing data in evaluations, but her focus is on using data for 
administrative purposes (e.g., evaluation design, sampling design), rather than in adopting items from an 
existing instrument. Sudman & Bradbum (1982, p. 14), on the other hand, encourage the questionnaire 
developer to examine existing instruments for possible Hems, both for technical reasons (e.g., 
established reliability) and the posstoflity of comparing results with s&nSar studies. 

The purpose of this paper is to explore on® context in which existing items may provide a convenient 
source of questions. More importantly, when property used, these items can significantly expand the 
usefulness and generailzability of the information that is collected, with Me or no increase in the total cost 
of data collection. @v sharing their experiences, the authors hope to raise the awareness of other survey 

1 Tfils paper is based on activates supported by the U.S. Department of Energy, Office of UnlvereBy and 
Science Education, and performed by Oak Ridge Associated Universities. Oak Ridge Associated 
Universities operates under the U.S. Department of Energy Contract Mo. DE-ACO5-76ORO0O33. Any 
opinion, findings and conclusions, or recommendations expressed in this paper are those of the authors 
and do not necessarily reflect the policies and views of the U.S. Department of Energy, Oak Ridge 
Associated Universities, or th@ University of Tennessee. 
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researchers about the potemiai advantages of adopting Kerns from existing instruments. Because we do 
not endorse such item adoption as a standard questionnaire design practice, however, we will discuss 
some of the factors that should be considered before making such a decision. 

EflOPflfitbffl 

M given a choice, most researchers would prefer to execute a true experimental design In conducting 
program evaluations. Such designs, if property conceptualized and carefully implemented, allow the 
researcher to reach dsrens&le conclusions about the existence and extent of programmatic effects. 
Unfortunately, many situations exist in which the use of a true experimental design is either Impossible or 
impractical These situations include such factors as ethical constraints (e.g.. problems of withholding 
positive treatments), sampling constraints (e.g., inability to randomly select anoVor randomly assign 
subjects to treatment groups), and the potential contamination of the treatment and control groups. 

In the context of evaluating the effects of national educational programs in a competitive (i.e.. grants 
and contracts) environment, the considerations are apt to be much more mundane. If a researcher 
proposes to implement an aggressive, empirically sound, true experimental evaluation design, his/her 
proposal would probably not be financially competitive with proposals that use less rigorous evaluation 
designs. The HkeChood of its Inherent scientific superiority being valued highly enough by the 
sponsoring agency to warrant Us being funded instead of a less expensive, less rigorous alternative, is not 
at all predictable. Often sponsors are (understandably) more concerned with showing the positive 
outcomes of a program (so that they can justify its continuation) than they am w&h determining which of 
these outcomes may be clearfy attributable to the program Because positive outcomes are often easy to 
Scfentify and frequently seem to be acceptable indicators of the worth of a program in the eyes of those 
who fund them, the program sponsors are often reluctant to commit funding to sophisticated evaluation 
procedures that could otherwise be spent on the program Bseff . 

The extent to which a professional ©valuator fe willing to develop and undertake a program 
assessment that is less scientifically rigorous than he/she might Bke Is a highly personal matter, but this Is 
an Issue that wS not be considered hem. The realities of contract evaluation often dictate that 
compromises be made In "best practice" techniques. This does not mean that oompetarvely-won 
evaluations are necessarily inferior evaluations. Nor does 8 mean that program effects cannot be 
established for evaluations thai do not use true experimental designs. The purpose of this paper is to 
offer suggestions, based on the experiences of the authors, on how evaluations undertaken In a 
competitive arena in which true experimental designs am not viable can be designed in such a way that 
meaningful comparative data can be examined. This approach is consistent with Patron's (tS82) notion of 
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practical evaluation, and Is responsive to Ws call for generating "a great deal of reaBy useful information 
wfth extremely scarce resources" (p. 19). 

Making Us* of Existing Comparison Groups: A Case Study 

Oak Ridge Associated Universities (ORAU) Is a private, not-for v rof8 corporation sponsored by 59 
colleges and universities and is a management and operating contractor for the U.S. Department of 
Energy. Through its Science/Engineering Education Division, ORAU conducts educational program 
evaluation and assessment studies for a variety of sponsors. In general, the objectives of the evaluation 
and assessment activities include providing quantitative and qualitative measures of the impact of these 
programs on participants, assessing programmatic achievements, providing information for the 
improvement of program operations, and determining the extent to which the programs meet their 
objectives. The educational programs are Mended to enhance some element of the production of 
scientists and engineers (S/Es) in this count ry-for example, the recruitment and retention of students to 
S/E programs of study and S/E careers; the pursuit of graduate S/E degrees; the extent of research 
involvement by scientists and engineers; or addressing the current underrepresentation of women and 
minorities in S/E study and S/E careers. 

One such program is the Science and Engineering Research Semester (SERS) program, which is 
sponsored by the Office of UnJversSy and Science Education, U.S. Departmert of Energy (DOE). The 
SERS program provides research appointments to about 150 U.S. undergraduate students each year at 
one of six DOE national laboratories. Participants have the opporturtfty to become involved in "hands-on" 
research, working with scientific teams engaged In long-range investigations and using state-of-the-art 
facilities and equipment SERS research appoemments are available In btomedk&ie, chemistry, materials 
science, engineering, physics, environmental science, geosdence, mathematics, computer science, 
artificial intelligence, energy systems, and waste technology. One the objectives of the SERS program Is 
"to encourage students to seek graduate degrees and research careers In science and engineering 
disciplines or areas supportive of the DOE mission" (Stevenson et a!., 1081. p. 8). 

Another such program 5§ the Laboratory Graduate Research Participation (Lab Grad) program. The 
Lab Grad program is an important component of the University/DOE Laboratory Cooperative Program, 
which Is also supported by the Office of University and Science Education in DOE. The program enables 
about 200 graduate students annuaRy to conduct thesis or dissertation research In residence at a DOE 
facility. Lab Grad partt&pants m>rk with laboratory §efent$fc staff and sr@ afete to make m® of equepmefEt 
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and facilities that are net generafly available ©n university campuses. Participants' research projects must 
complement ongoing research at the host facilities and must meet degree requirements for the students' 
graduate programs. Among the stated objectives for the Lab Grad program are "to encourage graduate 
students to pursue careers and to continue to work in areas supportive of the DOE mission* (VJvto & 
Stevenson, 1990, p. 4). 

Mflfld faf Comparison Groups 

The SERS and Lab Grad programs are typical of educational programs for which the authors have 
b/ n conducting follow-up assessments for several years. The problems inherent in evaluating the 
extent to which these programs meet general (but dearly primary) objectives such as those above are 
considerable. The measurement of short-term effects (e.g., pre- and post- research-experience 
attftudinal comparisons) are of very Med use, as far as determining whether participants wilt indeed go on 
to pursue graduate degrees or research careers In science or engineering. Long-term follow-up of former 
participants is essential to reaching firm conclusions about !he attainment of such ebfectivrts. The special 
difficulties In maintaining current addresses for students and early-career graduates are weO established. 
Assuming that some former participants do Indeed pursue graduate degrees and/or research careers In 
science or engineering, it is difficult to attribute those choices to the SERS or Lab Grad programs withe ii 
some basis for comparison. Identifying, surveying, and foltowing-up a control group of non-SERS .kmv 
I ab-Qrad students for each cohort of SERS or Lab Grad participants is economically infeasfote for the&e 
programs. 

MtettDcatfQD of Appropriate Comparison Groups 

In order to assess the degree to which programs lice SERS or Lab Grad have met these general 
objectives in the absence of traditional control groups, researchers at ORAL? and Argortne National 
Laboratory have turned to several national data bases containing information on scientists and engineers, 
in cooperation with the National Science Foundation (NSF) and the National Research Courted, 
questionnaires have been developed in which key items used In the evaluation of the programs being 
evaluated conform to these used in large national studies sponsored by NSF, DOE, and other federal 
agencies. Using questions and response options from these national surveys makes 8 possible to 
compare program participants to national norms wfh respect to many refsvai * variables. 

Depending on the objective being assessed and the point In time at which the assessment is mad® 
(relative to the completion of parttelpatfon In the educational program), Hems from on® or more of the 
following national surveys (for which NSF 5s the primary sponsor) are used: (a) the Survey of lamed 
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Doctorates, (b) the Survey of Doctorate Recipients, (e) the Survey of Recent Science, Social Science, 
and Engineering Graduates (a.k.a. the New Entrants Survey), and (d) the National Survey of Natural and 
Social Scientists and Engineers (a.k.a. the Survey of Experienced Scientists and Engineers or me 
Postcensal Survey). Each of these surveys is described below, based on the data collection procedures 
used in the 1980s. (The Science Resources Studies Division at NSF has undertaken a major 
restructuring of these data collection efforts for the 1990s, but all of the procedures for the new system 
are not yet in place.) 

SuOMX Pi Eaoffld Doctorates. This survey is conducted annually for NSF by the National Research 
Council to collect information on the number and characteristics of recipients of doctoral degrees In the 
United States. The data gathered in this survey are used to construct the Doctorate Records File (DRF), 
which is virtually a census of all recipients of research doctorates (exctocfing professional or clinical 
degrees such as the J.D., M.D., and D.V.M.) awarded by U.S. educational institutions since 1958. The 
DRF contains almost 1,000,000 records. (National Science Foundation |NSF), 1987, pp.15- 19; 
Thurgood & Weinman, 1989) 

gUBgy. of Doctorate Ractetenta . The Survey of Doctorate Recipients is one of three NSF surveys that 
cover various S/E subpopulattons. Together, these three surveys comprise NSPs Scientific and 
Technical Personnel Data System (STPDS). Conducted MenniaSy for NSF by the National Research 
Council since 1973, the Survey of Doctorate Recipients Is designed to provide national estimates of the 
supply and utilization of science and engineering doctorates. This longitudinal survey is based on a 
sample drawn from the DRF, and is stratified on several characteristics (source and type (S/E or non-S/EJ 
of degree, sex. fleW of doctorate, year of doctorate, raclaPethnlc identification, and citizenship). The 
sample size in 1987 was approximately 60.000. which represents about a 14n-13 sample of S/E 
doctorates. Data are collected for major demographic and empfcyment-related variables. Demographic 
variables include age, citizenship, marital status, sex. race, and ethnicity. Employment-related variables 
include employment status, emptoyment sector, primary work acSvay, and salary. (Holmstiom, 1988; NSF. 
1987, pp. 21-25) 

Maw Efftranis Suavey . A component of the STOPS, the objective of this cross-sectlonaJ biennial 
survey, conducted for NSF by the Institute for Survey Research at Temple University, is to provide data on 
the demographic and emptoyment characteristics of indMduats who receive bachelor's or master's 
degrees In S/E fields from U.S. institutions. The survey population is limited to S/E degree recipients who 
were citizens or permanent resident aBens at the time of their degree award. A two-stage probability 
sample Ss used, with the primary sampling unit being universities and colleges, stratified by geographic 
region, public/private institutional status, type of curriculum offered, proportion of graduates with S/E 
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majors, and two special strata consisting of universes and colleges that have a predominantly ©Jade 
student body or that have high concentrations of Wspantas. The secondary sampling unit consists of 
graduates drawn from the sample of universities and coBeges. individuals drawn from Institutions in the 
special strata are oversarnpled In order to increase the reliability of data on racial/ethnic groups. The 
sample size for the 1986 survey, which Included the graduating classes of 1982, 1984, and 198$, was 
about 36,000. LBte the Survey of doctorate recipients, data are collected for major demographic and 
employment-related variables. Demographic variables include age, citizenship, marital status, sex, race, 
and ethnicity. Early career employment-related variables Include employment status, employment sector, 
primary work activity, and salary. (Kolmstrom, 1988; NSF, 1987, pp. 27-39) 

Susvay of Expaftencad Scientists and Enpinaers. An integral component of the STPDS, this biennial 
longitudinal survey provides data on the number and characteristics of Individuals who were identified as 
being part of the S/E population during the preceding decennial census. The original (I.e., based on the 
1980 decennial census) sample of about 138,000 people was stratified on the basis of education, 
occupation, sex, and race, information is collected on education and training (level and field of degree), 
demographic characteristics (age, citizenship, marital status, sex, race, handicapped status, and ethnicity), 
employment status, and employment profile (occupation, type of employer, primary work activity, salary, 
work experience, etc.). (NSF, 1987, pp. 3-13) 

Satecten of Items 

Only those items from the "parent" surveys that are dearly relevant to the objectives of the program 
being assessed are selected for Inctuiten In the instruments for the programs being assessed. These 
may Include general demographic items (e.g., gender, race, citizenship) as weft as items that relate to 
specific program-related characteristics (e.g., academic major, degree level, employment specialty). 
Demographic differences between the two survey populations may result In rival hypotheses about 
program effects, so it Is useful to have the same response options in both instruments ff at an possible. 

The comparisons to be made from the use of such existing data clearly cannot generate conclusions 
as definitive as those drawn from a tn?® experimental design. In order to Increase the relevance of these 
comparisons, nevertheless. It & important to minimize the mmkm and severity of uncontrolled Influences 
on the comparisons. The careful selection and construction of items on the new instrument Ea critical *o 
maintaining the credMy and usefulness of the comparisons. 

Figures 1 and 2 are examples of seemingly straightforward demography items that vary from one 
survey to another. Because respondents are Influenced by the choices thsy are offered, K is Important 



that the researcher pay attention to the response options used in the data collection instrument from 
which the comparison group information came, in the case of items for which the ciiaracterfstics being 
measured may be rotated to the expected outcomes of the evaluation (e.g., when rival hypotheses 
emerge), the evaluator should adopt the same item wording and response options whenever possible. 
This means that he/she win have to have already identified the comparison groups) and the desired 
comparisons when the evaluation instruments are constructed. 

Figures 3, 4, and 5 are examples of items from the NSF-sponsored m eys that have been integrated 
Into follow-up assessments of OOE-sponsored programs fke SERS &«? lab Grad. Although the iype-cf- 
emptoyer Hem (Figure 3) is rather general in nature, it was important to oSe? the same re nse options as 
the national surveys in order to make legitimate comparisons between ths DOE participants and broader 
groups of scientists and engineers. The prtrnary-woifc-actfVity Hem (Figure 4) and the area-of-nattonaS- 
interest item are more specific and are intimately tied to the objectives of DOE-sponsored educational 
programs. 

Designing and Interpreting Comparisons 

Once the data are collected from former program participants using items drawn from the NSF 
questbnnaire(s), the precision of the comparisons desired (or the precision that is possfcte) determines 
how the comparison data are derived and presented. In some cases, only very general tJenchmaite" are 
needed In order to permit a reader to achieve a reasonable perspective on reported characteristics of 
program participants. It may not matter that the groups are not strtcify comparable. Punished data often 
suffice for this purpose. When more precise comparisons are desired (and both data sets and survey 
methodologies suggest its appropriateness), special tabulations are requested from the NSF survey 
contractor or special tabulations are generated by ORAU from a DOE-ewned sponsors' data tape. 

Benchmark comparisons . Figures 6, 7, and 8 are examples of comparisons drawn from published data 
and/or spedal analyses of national data sets that are directly related to the objectives of DOE educational 
programs. In Figure $, the primary work activity of Lab Grad respondents is compared to various groups of 
scientists and engineers. Ail but une of these comparisons are drawn from pubfished data One has only 
to consider the difficulty of interpreting-even in the most general terms-the data from former Lab Grad 
participants in the absence of the other columns in order to recognize the value of these nonequivaient 
comparisons. Even though each of the comparison groups is not comparable to the former Lab GradSs in 
at least some respect (e.g.. age. degree field, degree level, year surveyed), the distribution of work 
activities of these groups are nevertheless instructive in gauging the work activities of the Lab Grade. Th© 



comparisons In Figures 7 and 8 likewise provide a valuable backdrop against which the Lab Grads' 
Involvement in federally funded work can be viewed. 

Mom definitive comparisons , m? foltow-up assessment of former participants In the SERS program 
provided opportunities for deriving more precise comparisons by using the data tape of the 1988 Mew 
Entrants Survey. The 1988 survey Includes graduates of the classes of 1985-1986 and 1986-1987. For 
the purposes of the SERS report, a special tabulation was performed using only those students earning 
bachelor's degrees In science or engineering fields during the academic year 1986-1987. 

Data for the 1988 New Entrants Survey was collected in the spring of 1988, so the 1986-1987 
graduates would have been out of school for approximately one year, likewise most individuals who were 
SERS participants as Juniors in the academic year 1987-1988 and those who were SERS participants as 
seniors In the academic year 1988-1989 would have been out of school for approximately one year in the 
fan of 1990 when data were collected for the fofow-up assessment. As a result, mis comparison, aShouyh 
imperfect, allows one to see how former participants In the SERS program differ from science and 
engineering students who received their bachelor's degrees at about the same time. 

Figure 9 depicts a comparison of the academic status of former SERS participants and 1987 SfE 
bachelor's degree recipients. Even though some of the SERS participants had not received their 
undergraduate degrees (meaning that the proportion who ultimately attend graduate school Is an 
underestimate in the figure), me extent to which former SERS participants were pursuing graduate 
degrees was dearly remarkable compared to the 1987 bachelor's recipients. This tendency is directly 
relevant to the objectives of the SERS program. Likewise the undergraduate degree fields of the SERS 
participants are more consistent with those subsumed under DOE'S mission than are those of S/E 
bachelor's recipients in general (see Figure 10). 

Making Use of Existing Comparison Groups: Other OR All Examples 

in the evaluation of a selective program for high school science and mathematics teachers, there was 
no control group. Snstsgd P eonparisons were sought fmm existing research and data bases. Professional 
organisations, such as the National Science Teachem Association (NSTA), provide one source of 
comparative data. NSTA annuaBy surveys teachers who taught one or more sections of science In pufeSc 
or private high schools in the United States. Information includes such information as the number of 
sections of science or math taught, gender, school type, level, and school stee. 



The National Carter for Education Statistics, under the Office of Educational Research and 
Improvement of the U.S. Department of Education, annually compiles a Digest of Education Statistics, 
based on Information from several sources, Including surveys and activities of the Center. The 1990 
report has one chapter containing over 100 pages of tables on elementary and secondary education. The 
Center annually surveys public schools and periodically surveys teacher characteristics. Summary data on 
students, staff, and finances are provided by states. Data are also Incorporated on achievement (from the 
National Assessment of Educational Progress), racial/ethnic enrollment (from the Office of Civil Rights), 
teacher characteristics, teaching assignments, and average salaries (from the National Education 
Association and the American Federation of Teachers), private school enrollment (from the National 
Catholic Educational Association), mandatory ages of attendance, requirements for graduation, and 
minimum competency testing for students and teachers (from the Education Commission of the States), 
scores on the Scholastic Aptitude Test for college-bound seniors (from the College Entrance Examination 
Board), and the proportion of high school graduates who go on to college (from the Bureau of Labor 
Statistics). In addition to totals, data for some Hems are presented by state and/or broken down by other 
categories. 

Results of other large-scale or targeted surveys can be located by searching the cataloging systems of 
the Educational Resources Information Center (ERIC), PSYCHUT. or others. Such a search helped 
identify articles about teachers of exemplary science programs (Penk*. Yager. & Bonnstetter. 1986). The 
articles on exemplary programs referenced (for a comparison population) a national survey of science, 
mathematics, and social studies education conducted under the auspices of the National Science 
Foundation (Weiss, 1978). It was possfete to compare characteristics of program participants with those 
who were identified as having exemplary science programs. 

The literature search also helped to locate research stiKfles thai had used Instruments for measuring 
variables of interest In the study. Two other sources for nor .commercial Instruments in science education 
are handbooks published by the ERIC Science. Mathematics, and Environmental Education 
Clearinghouse (Mayer, 1974; Munby. 1983). While standardized achievement tests for students are 
available, instruments for measuring attitudes of teachers related to the program were more difficult to find. 
When the most seunrSy developed instrument related to project outcomes was selected (Thompson & 
Shrigfey, 1988). there were no norms for comparison. The instrument was, however, used on a pre- and 
post- basis Id measure changes in participant attfSudss. 
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Conclusions 



The techniques described !n this paper clearly do not represent a panacea tor evaluation IBs. The use 
of existing data sources for comparison groups is rteftner helpful nor advisable in many situations. Lifting 
ftems from existing Instruments Is not a sutrttoite for thoughtful Instalment development In cases In 
which 8 makes sense to compare program participants to well-defined external populations or to general 
populations, however, researchers might do weB to consider using these techniques In planning their 
evaluations. To maximize the taterpretabffitv of the results, the comparison group-together with the 
questionnaire^) that produced the data associated with that group-must be identified and evaluated 
before assessment instruments are designed This requires thai the researcher not only acquire the 
Instrument but that he/she have a thorough understanding of me methodology (e.g., sampling frame, 
sample design, survey and analysis techniques, response rate, norvesponse adjustments) of the data 
collection effort Once the researcher understands the characteristics of the potential comparison group, 
he/she must consider the limitations of the design nuances and the content of the Rems themselves with 
regard to the validity, robustness, and relevance of any comparisons that might be made. Property 
handled, however, the use of existing data for comparison purposes can be a useful and cost-effective 
alternative to traditional control-group evaluation designs. 
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Figure 1. Examples of Variation In Response Options for 
Demographic Items: Race/Ethnlelty 



from Survey of Earned Doctorates, 1989-90 

9. What is your racial background? (Ctexk ©nfy om.) 

0 □ American ImSan or Alaskan Native 

1 □ Asian or Pacific islander 

2 □ Stock 

3 O White 

10. Are you Hispanic? O No □ Yes — > 0 □ Mexican American 

1 O Puerto Rican 

2 □ Other Hispanic 



from 1990 Survey of Natural and Soda! Science ami Engineering Graduates 

29. Are you: 

1. O American Indian or Alaskan Nailv© 

2. □ Asian or PacSfc islander 

3. 0 Blade 

4. □ White 

5. O Other. PLEASE SPECIFY 



from 1990 Information Sheet for Participants in the US. Department of Energy 
Science and Engineering Research Semester Program 

3. What is your racial background? 

1. □ American Indian or Alaskan Native 

2. O Asian 

3. O Pacfite islander 

4. □ Black 

5. □ Caucasian (Not Hispanic) 

6. G Hispanic 

7. O Othar 
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Figure l(eontinued). Examples of Variation In Response 

Options for Demographic Items: 
Race/Ethnlclty 



from National Education Longitudinal Study of 1988 (Eighth Grada 
Questionnaire, NELSM) 

31 A. Which best describes you? (MARK ONE) 

Asian or Pacific Islander — > GO TO 31B below 

Hispanic, regardless of race — > GO TO 31C be tow 

Black, not of Hispanic origin | 

White, not of Hispanic origin |— > SKIP TO Question 32, Pag® IS 

American Indian or Alaskan Native j 

3 IB. Which of these best categorizes your background? (MARK ONE) 

ASIAN OR PACIFIC ISLANDER 

Chinese 
Filipino 
Japanese 
Korean 

Southeast Asian (Vietnamese, Laotian, CarrfeocSarVKampuchean, Thai, etc.) 

Pacific Islander (Samoan, Guamanlan, etc.) 

South Asian (Asian Indian, Pakistani, Bangladeshi, Sri Lankan, etc.) 

West Asian (Iranian, Afghan, Turkish, etc.) 

Middle Eastern (Iraqi, Israeli, Labanese, etc.) 

Other Asian 

31 C. Which of these best categorizes your backgrourtd? (MARK ONE) 
HISPANIC 

Mexican, Mexican- American, Chteano 
Cuban 
Puerto Rican 
Other Hispanic 

31D. What Is your rase? (MARK ONE) 

Black Hispanic 
Whfle Htopante 
Other Hispanic 
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Figure 2. Examples of Variation In Response Options for 
Demographic Items: Citizenship 



froto Survey of Earned Doctorates, 1989-90 

7. Citizenship: 

□ United States, native 

O United States, naturalized 
Non-United States 

□ Permanent Resident of UnSed States {Immigrant visa) 

Q Temporary Resident of UnSsd States (Non-Immigrant visa) 



from 1990 Survey of Natural and Social Science and Engineering Graduates 

26. Are you: 

1. O U.S. citizen 

2. □ U.S. naturalized 

3. □ Non-US. Immigrant 

(Permanent Resident) 

4. O Non-U.S. cftizen, nonimmigrant 

(Temporary Resident) 



from 1989 Information Sheet «or Former Participants in the U.S. Department of 
Energy Laboratory Graduate Thesis Research Program 



5. Citizenship 

1. O U.S. Native Bom 

2. G U.S. Naturalized 

3. G Permanent Resident Alien 

4. G Non-U.S. 



Fgure 3. Example of Stem Adopted for Use in Focused Follow-Up 
Assessment: Type of Employer 



7. Which category best describes the type cf your principal employment OR postdoctoral appointment during 
FEBRUARY 1989? 



0 Self-employed 

1 Business or industry 

2 Junior college, 2-yoar college, technical institute 

3 Medical school (including unJvQrsfty-afffflaled hospitaJ or medteal comer) 

4 . 4-year coBege 

5 „ University, other than medical school 

6 Elementary, middle, or secondary school system 

7 . Private foundation 

8 Hospital or cilnJc 

9 U.S. military service, active duty, or Commissioned Corps, e.g., USPHS, NOAA 

10 . U.S. government, civilian employee 

11 State government 

12 Local or other government, specify 

13 Nonprofit organization, other than those listed above 

1 4 Other, specify — — — 



Source: 1989 Survey of Doctoral Scientists and Engineers 
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Figure 4. Example of Item Adopted for Use In Focused Follow-Up 
Assessment: Primary Work Activity 



12. From the activities listed below, select your primary and secondary work activities for your principal job (as 
reported In #6), In teims of time devoted during a typical week. 

Enter the appropriate codes (1-16) for each in the specified space. 

. Primary activity Secondary activity 



1. Teaching 

2. Basic research (Le., study directed toward gaining scientific knowledge primarily for its own 

sake) 

3. . Applied research (i.e., study directed toward gaining scientific knowledge in an effort to meet 

a recognized need) 

4. Development of equipment, products, systems 

5. Design of equipment processes, models 

6. ._ Management/administration of R&D 

7. Management/administration of educationaVotnsr programs 

8. Report and technical writing, editing 

9. Professional service to individuals, ctinical diagnosis, psychotherapy 

10. Consulting 

1 1 . Operations-production, maintenance, construction, Installation 

12. . Quality control, testing, evaluation 

13. , Sales, marketing, purchasing, customer and public relations 

14. Statistical work-survey work, forecasting, statistical analysis 

15. Computer applications 

16. „_ Other, specify: 



Source: 1969 Survey A Doctoral Scientists and Engineers 
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Figure 5. Example of Item Adopted for Use In Focused Follow-Up 
Assessment: Area of National Interest 



1 8a. From this list of selected areas of national interest, indicate the ONE area to which you devoted the MOST 
professional time during a typical week at the Job reported in #6. 



1 . Energy and fuel 

2 Health 

3 __ Environment 

4 Education 

5 National defense 

6 Food or agriculture 

7 Biotechnology 

8 _____ Mineral resources 

9 Community development and service 

10 Housing (planning, design, construction) 

11 Transportation 

12 Communications 

13 Space 

14 None of the above 



Source: 1989 Survey of Doctoral Scientists and Engineers 
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Figure 6. Example of Benchmark Comparison Groups: Primary Work Activity 



Primary Work Activity of 1970-1967 Lao Grad Survey Respondents and Other Scientists 
and Engineers 





Lab Grad 


PhD&Eo 




DOE- 




Energy- 


Activity 


dent* 


Of Ag0 


Ait ot.n 
AurnD 


Funded 
PhD$m# 




Rotated 
&<Et* 


Applied Research 


19.6% 


23.9% 


17.2% 


31.2% 


5 6% 


7 3% 


Basle Research 


26.7% 


22.4% 


15.1% 


21.8% 


2.9% 


2.6% 


Development/Design of 


3.5% 


5.2% 


4.5% 


4.0% 


18.9% 


24.8% 


Products/Processes 












SUBTOTAL -R&D 


49.$% 


51.5% 


36.8% 


57.0% 


27.4% 


34.7% 


Computer Applications 


3.9% 


3.1% 


2.8% 


1.4% 


10.2% 


8.8% 


Consulting 


8.6% 


2.3% 


3.3% 


5.0% 


5.5% 


7.4% 


Management of Programs or R&D 


5.1% 


8.8% 


16.2% 


17.9% 


27.7% 


19.1% 


Professional Services to Individuals 


3.9% 


10.3% 


7.8% 


0.2% 


1.9% 




Teaching 


7.1% 


18.8% 


26.2% 


12.5% 


7.7% 


13% 


All Others and No Response 


21.6% 


5.2% 


6.9% 


6.0% 


19.6% 


28.7% 5 


TOTAL 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 



NOTES: 

1 Characteristics of Doctoral Scientists and Engineers in the United States: 19B7, NSF 88-331 
2 ORAU data based on NSPs 1987 Survey of Doctorate Recipients 
3 ttS. Scientists and Engineers: 1886, NSF 87-322 

4 Energy-Rsiated Science and Engineering Personnel Outlook, 1987; DOE/OR/00033-H1 
5 lnckfdes Professional Sendees to Individuals 



Source: U.S. Department of Energy Laboratory Graduate Research Participation Program: 
Profiie ami Survey of 1979-1987 Participants, April 1990, pp. 19-20. 




Figure 7. Example of Benchmark Comparison Groups: Status of Federal 
Support 



Status of Federal Support for 1079-1987 Lab Gred Survey 
Respondents and Other Scientists snd Engineers 



Status of F&deral Support 


LabGrad 
Respon- 
dents 


All PhD 


All SGs* 


Energy- 
Related 
PhD&Es* 


Funded 


61.9% 


43.7% 


30.9% 


42.2% 


Not Funded 


31.4% 


52.7% 


64.2% 


56.7% 


Net Sure and No Response 


$.7% 


3.6% 


4.9% 


1.1% 


TOTAL 


100.0% 


100.0% 


100.0% 


100.0% 



NOTES: 

1 Characteristics of Doctoral Scientists and Engineers in the United 
States: 1997, NSF 88-331 

2 US. Scientists and Engineers: 1986, NSF 87-322 

3 Doctoral Scientists and Engineers Working in Energy-Related 
Activities, 19BS; OOE/ER-0322, May 1987 



Source : US. Department of Energy Laboratory Graduate Research Participation Program: 
Profile and Survey of 1979-1987 Participants, April 1990, p. 21 . 
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Figure 8. Example of Benchmark Comparison Groups: Federal Funding 
Source 



Federal Funding Source of 1979-1987 Lab Grad Survey Respondents end Other 
Scientists end Engineers 



Source of Federal Funds 


L&bGrad 
nespofr' 

dents 




All 


Energy- 
Related 
PhD&Es* 




Department of Defense 


17.1% 


23.6% 


40.0% 


17.4% 




Department of Energy 


51.3% 


13.5% 


7.6% 


55.1% 




Environmental Protection Agency 


8.2% 


4.3% 


3.8% 




5 


National Institutes of Health 


11.4% 


24.4% 


5.9% 3 




5 


National Science Foundation 


10.1% 


17.2% 


3.1% 


11.1% 




All Others 


31.6% 


47.5% 


31.6% 


49.6% 


6 


Uncertain of Support/Don't Know Source Agency 


1.3% 


6.0% 


13.8% 






TOTAL 


131.0% 


136.5% 


105.8% 


133.2% 





NOTES 

Percentages are the proportion of funded S/Es who receive support from each agency. 
Totals add to more than 100% because some individuals are funded by more than one agency. 

1 Characteristics of Doctoral Scientists and Engineers In the United States: 1967, NSF 88-331 

2 U S. Scientists and Engineers: 1986, NSF 87-322 
^Includes all of DHHS 

4 Doctoral Scientists and Engineers Working in Energy-Related Activities, 198$; 

DOE/ER-0322, May 1987 
s !nduded In All Others category 
^Includes Doni Know Source Agency 



Source: U.S. Department of Energy Laboratory Graduate Research Participation Program: 
Profite and Survey of 1979-1987 Participants, April 1990, p. 22. 
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Figure 9. Example of More Definitive Comparison Groups: Academic Status 



Academic Statue of 
SERS Respondent* 

(n«317) 




3.1% 30.0% 

Source: SERS follow-up survey of panic i pants during academic year 1 987- 1 988 through 
Fall 1989. 



Academic Status of 1986-1987 
Science and Engineering 
Bachelor's Graduates 




Not a Student 
70.7% 



Scuice: 1 988 NSF New Entrants Survey. 



Source: U.S. Department of Energy Science ana Engineering Research Semester: 
Profile and Survey of 1 $$7-1990 SERS Participants, July 1991 , p. 18. 
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Figure 10. Example of More Definitive Comparison Groups: Undergraduate 
Majors 



Undergraduate Science and Engineering Majors: 
SERS Participants Versus Bachelor's Graduates 



Selartfld Raids 



Biological 
Sciences 



Chemistry 

Computer/ 
Math. Set. 

Earth/Env. 
& Marine 
Sciences 

Engineering 



Physics/ 
Astronony 



SERS Participant* 
Bachetof** Graduates 




10% 



— j j j , 1 (- 

20% 30% 40% 50% 
Percent Distribution 



Note: These data have been restricted to induce enfy engineering and 
physical, biological, and cc^putef/mathernatical sciences. The 
figure represents distribution across these selected fields. 

SERS participants include those for academic years 1987-1988 
through 1989-1990. 

Bachelor's graduates data were from the 1988 New Entrants Survey 
for those who received deo'ees in the academic year 1986-1987. 



Source: U.S. Department of Energy Science andEngi 

of 198 



ineering 

1990 SERS Participants, July 1091 , p. 1 
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